User interface for efficiently displaying relevant oct imaging data

ABSTRACT

The present invention is an OCT imaging system user interface for efficiently providing relevant image displays to the user. These displays are used during image acquisition to align patients and verify acquisition image quality. During image analysis, these displays indicate positional relationships between displayed data images, automatically display suspicious analysis, automatically display diagnostic data, simultaneously display similar data from multiple visits, improve access to archived data, and provide other improvements for efficient data presentation of relevant information.

PRIORITY

This application is a continuation of U.S. application Ser. No.14/245,910, filed on Apr. 4, 2014, which is a divisional of U.S.application Ser. No. 13/549,370, filed Jul. 13, 2012, which is adivisional of U.S. application Ser. No. 11/978,184, filed Oct. 26, 2007,now U.S. Pat. No. 8,223,143 both of which are incorporated by reference.This application claims the benefit of the filing date under 35 U.S.C.§119(e) of Provisional U.S. Patent Application Ser. No. 60/854,872,filed on Oct. 27, 2006, and Provisional U.S. Patent Application Ser. No.60/857,451, filed on Nov. 7, 2006, which are hereby incorporated byreference in their entirety.

ACKNOWLEDGEMENT OF GOVERNMENT SUPPORT

Portions of this disclosure were developed with Government support underGrant No. 6 R44EY014099-0, awarded by the National Institute of Health.The Government may have certain rights in the claimed inventions.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to optical medical imaging, and inparticular to systems that are adapted to perform Optical CoherenceTomography (“OCT”) for use in diagnosis and monitoring of tissue health.

BACKGROUND OF THE INVENTION

Optical Coherence Tomography (OCT) is a technology for performinghigh-resolution real time optical imaging in situ. OCT herein refers toany of the transverse scanning extensions of one-dimensional opticalcoherence detection techniques generally derived from optical coherencedomain reflectometry (OCDR) or optical frequency domain reflectometry(OFDR). OCT is an optical measurement and imaging technique usinglow-coherent light from a broadband source or a tunable laser to createinterference signals across the tunable wavelength range of the laser toilluminate both a reference path and a sample path. The superposition ofbackscatter reflection from the sample path and the optical signal fromthe reference path creates an interference pattern. The interferencepattern contains information about the scattering amplitude as well asthe location of the scattering sites in the sample. The longitudinalrange within the sample is obtained by using time domain or frequencydomain optical coherence techniques. This depth profile is commonlycalled an “A-scan”. Cross-sectional images are synthesized by laterallyscanning the sample beam over a series of adjacent A-scans, 2-D and 3-Dimage scanning. OCT provides a mechanism for micrometer resolutionmeasurements.

Evaluation of biological materials using OCT was first disclosed in theearly 1990's (see U.S. Pat. No. 5,321,501). More recently it has beendemonstrated that frequency domain OCT has significant advantages inspeed and signal to noise ratio as compared to time domain OCT (Leitgeb,R. A., et al., Optics Express 11:889-894; de Boer, J. F. et al., OpticsLetters 28: 2067-2069; Choma, M. A., and M. V. Sarunic, Optics Express11: 2183-2189). In Spectral Domain OCT (SD-OCT), sometimes also referredto as Frequency Domain OCT (FD-OCT), and also sometimes also referred toas Spectral Radar (Optics letters, Vol. 21, No. 14 (1996) 1087-1089),the measurement is achieved by examining the spectral content of theinterference pattern out of the interferometer.

Improvements in imaging displays frequently accompany improvements indata acquisition methods and devices. For example, development of higherresolution imaging devices creates a need or motivation for higherresolution imaging displays; faster 2-D data acquisition increases theneed for high speed data transmission and storage and motivatesimprovements in 3-D display applications; improvements in the signal tonoise ratio in acquired data stimulates new uses and displays for thatinformation.

Large medical imaging data sets, such as those acquired duringvolumetric imaging, present difficulties in displaying relevantinformation to operators/users. Medical practitioners need to obtainrelevant information quickly in a format that can be efficientlyprocessed. A traditional approach to displaying 3-D volumes ismulti-planar reconstruction, which simultaneously displays images fromdifferent viewing angles. The user then “scrolls” through the volumelooking for relevant images. An alternative approach utilizes moderncomputational power to identify features of interest and present theseto the user through volume rendering. Many times, however, an expertuser benefits from observing individual slices of the image datadirectly. However, selection of these images can be time-consuming andthere is a need to improve the means for accessing relevant slices.Herein, a volume slice will generally refer to planar data extractedfrom a volume, while B-scan will refer to a planar section of the volumethat was acquired sequentially. In this sense, a B-scan is a slice,while a slice may be a B-scan. However, the terms are often usedinterchangeably in the literature and the distinction is often notrelevant, since a slice could have been a B-scan under an alternativescanning sequence.

Increased longevity within the population increases the likelihood ofage related conditions, such as macular degeneration and glaucoma. Lossof vision, whether partial or complete, dramatically affects quality oflife. Whether vision loss is due to changes in the anterior, posterior,or interior of the eye, monitoring change can be crucial to modernpatient management.

Change analysis is the detection of change in the condition of a patientover time. Change analysis has great potential for improving patientcare in areas such as diagnostic monitoring, intervention planning, andprogress monitoring. Modern computing and digital imaging make itpossible to store and retrieve large quantities of patient imaging data.Taking diagnostic advantage of these large quantities of data requiresimprovements in access and management of diagnostic combinations ofimaging data within an analysis package. For many diseases, thereremains an active debate over what should be measured and tracked overtime to track and/or predict disease progression.

Glaucoma is a term generally referring to the collection of diseasesrelated to loss of retinal ganglion cell function. Glaucoma is a slowlyprogressive disease that, unless treated (and sometimes even whentreated), can result in blindness. While raised intraocular pressure(TOP) is a symptom within a sub-family of these diseases, one patient'sdamaging TOP may well be completely tolerated by another patient with nodiscernable visual affects. (See U.S. Pat. No. 7,084,128, Yerxa, et al.,“Method for reducing intraocular pressure”) Glaucoma ProgressionAnalysis (GPA) software developed with Carl Zeiss Meditec by Dr. AndersHeijl represents the current state of Progression Analysis for Glaucoma.This software monitors visual field loss progression by examining thepatient's response to visual field stimuli over time.

Macular degeneration describes a disease or family of diseases that arecharacterized by a progressive loss of central vision. Vision loss isgenerally associated with abnormalities in the choroid, Bruch'smembrane, the neural retina and/or the retinal pigment epithelium.Destruction of a vascular function within the choroid depletesnourishment to retinal layers and damages overall visual function. Sincesuch destruction is, at present, not generally repairable, recognitionof the vascular failure frequently comes too late to be of any realvalue to the patient. Retinitis and retinopathy are retinal degradationsthat may progress into total loss of vision. Tracking the change(progression or regression) of eye function both prior to and posttreatment improves diagnosis and treatment. Tracking changes over timeimproves the timing of intervention and enables more effective patientmanagement.

In light of the above, there is a need in the art for an efficientmethod and apparatus designed to provide to the user relevant imagedisplays and analysis of the large data sets associated with volume OCTimaging. There is a need to display the relevant images needed to trackchanges over time. The present invention meets the need to providerelevant image displays to the user, overcoming past obstacles byimproved data presentation.

SUMMARY

The scope of the present invention is defined by the claims that follow.Nothing in this section should be taken as a limitation on those claims.

In accordance with one aspect of the present invention, the imagingsystem displays a small sample of image data in real time prior tovolume data acquisition enabling the user to align the imaging systembefore acquiring a full volume image.

In another aspect of the present invention, the imaging system processesa small sample of image data and automatically aligns the system beforeacquiring a full volume image.

In another aspect of the present invention, on a patient's second orlater exam, the medical provider can retrieve imaging data from one ormore previous exams, register imaging data across multiple visits anddisplay image data from two or more exam visits at the same time.

In another aspect of the present invention, navigation through one setof image data automatically navigates and displays equivalent image datafrom another exam.

In yet another aspect of the present invention, image movies played fromone exam are synchronized and registered to display the correspondingregion in another exam so that the change in image data can be readilyascertained. In one instantiation of this aspect of the presentinvention, the time scale for navigating through the image movie isnon-linear.

In yet another aspect of the present invention, the imaging system usesa small sample of image data to align the system automatically beforeacquiring a full volume image.

In yet another aspect of the present invention, user navigation throughone image dataset is registered with another exam so that displayedanalysis images from both exams display corresponding data.

In yet another aspect of the present invention, user modification of aboundary in a single image is propagated throughout a sequence ofimages.

In yet another aspect of the present invention, a summary image isdisplayed alongside of an OCT image slice and the location of the OCTimage slice within the volume is displayed in the summary image.Alternatively, an analysis image can be derived from the OCT volume dataand displayed, overlaid over the summary image.

In yet another aspect of the present invention, thumbnails are combinedto form a combination thumbnail, which can be used to identify and/orretrieve the exam.

In yet another aspect of the present invention, at least one image ofthe display contains a confidence map. The confidence map is indicativeof the confidence in the segmentation performed either on slice orvolume data.

In yet a further aspect of the present invention, software automaticallyidentifies the most relevant images, such as specific B-scans orarbitrary slices, and displays them to the user as an image.

The analysis of the change over time of physical attributes is awell-known diagnostic tool. Herein is provided a method and apparatusproviding a user interface for efficiently displaying relevant OCTimaging data.

BRIEF DESCRIPTION OF THE DRAWING

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates a user interface for aligning tissue for dataacquisition.

FIG. 2 illustrates the use of a color slice locator to associate thelocation of a volume slice within the summary image viewport.

FIG. 3 illustrates the use of color slice locators within the summary(fundus) window and also within each of three perpendicular volumeslices.

FIG. 4 illustrates a user interface for starburst scan data acquisition.

FIG. 5 illustrates additional slice location information that can bedisplayed in the fundus viewport.

FIG. 6 illustrates measurements on a thickness map.

FIG. 7 illustrates a networked system that can store and retrieve examdata either internally, or across a network from another system or froma data server.

FIG. 8 illustrates an exam retrieval interface with a combinationthumbnail containing a fundus image with analysis overlay, and en-faceimage with scan-type icon overlay and two thumbnail B-scan images usedfor exam identification.

FIG. 9 illustrates system alignment controls located in the Irisviewport next to the image display.

FIG. 10 illustrates system controls located in the fundus viewport nextto the image display.

FIG. 11 is a flow diagram illustrating a method of registering one imageto another.

FIG. 12 illustrates a user interface display for thickness maps. Thethickness map is displayed, along with the upper and lower surfaces usedto compute it. The thickness map is overlaid on an LSO map and multiplevolume slices of the OCT volume scan are displayed.

FIG. 13 illustrates an en-face overlay over an LSO image.

FIG. 14 is a flow diagram of an improved volume slice presentationassociated with lesions.

FIG. 15 is a flow diagram of a process for automatic propagation ofsegmentation modifications.

FIG. 16 is a flow diagram of a change analysis process for displayingregistered images from different exams.

FIG. 17 illustrates an analysis display for visualizing change.

DETAILED DESCRIPTION

The present invention is a User Interface (UI) efficiently providing theuser with relevant OCT image displays. In one instance, the UIsimultaneously displays images of the same region acquired duringexaminations performed at separate visits. Such displays enable theservice provider to monitor changes in the patient's condition overtime. The User Interface disclosed is useful for acquiring data,reviewing acquired data, simultaneously viewing multiple images, andmanipulating analysis displays. The User Interface provides access toanalysis applications that identify regions of interest, reduce thedata, and display relevant information in an efficient manner. The UserInterface uses image overlays to increase information density in adisplay area with minimal impact to the underlying display. Overlayshelp the user find, understand the location of, and visualize relevantdata. Image thumbnails and composite image thumbnails are used toreadily recognize (and optionally retrieve) exams from which they werederived. This User Interface has been implemented in conjunction with animaging system described in co-pending U.S. patent Ser. No. 11/820,773,filed Jun. 20, 2007, (published as US 2007/0291277) incorporated hereinby reference. However, said User Interface can, for many of itsfunctions, perform equally well on a stand-alone platform with access toOCT data files.

Patient Alignment

Optimal patient imaging requires proper patient alignment. The UserInterface can only function to help proper patient alignment whenrunning on the image collection system. FIG. 1 illustrates a userinterface display for aligning patients. For retinal exams, properalignment is achieved when the target region of the retina is centeredand focused, the retinal arc is centered in the central horizontal andvertical slices, and the maximal extent of the retina is visible withinthe volume cube. Ostensibly, there are three steps to patient alignment:aligning the patient's head in front of the ocular lens of theinstrument so that the working distance is correct (that is, the imagingsystem is aligned properly with the pupil), aligning the optics tocorrect for refractive error, and aligning the OCT imager to image atthe correct depth. A fourth step, polarization compensation betweenreference and source arms in the OCT imager to improve image quality,can also be performed, but it does not require changing the focal pointof any portion of the imaging system with respect to the patient.

The first step in proper patient alignment for the imaging system ofU.S. patent Ser. No. 11/820,773 is aligning the patient's head in frontof the ocular lens so that the working distance is correct. In FIG. 1,the upper left viewport 100 displays an Iris Viewer image 120. The IrisViewer image is a high contrast image of the iris surrounding a centraldark pupil. This image is used for aligning the patient workingdistance. Icon 110 overlays the Iris Viewer image showing the entrypoint of the scan beam and is used for aligning the pivot point of thescan beam on the subject's pupil. In the illustrated embodiment, icon110 is in the form of a cross-hair target. The Iris Viewer described inU.S. patent Ser. No. 11/820,773 and motorized chin rest described inco-pending U.S. patent Ser. No. 10/843,767, filed May 12, 2004,Publication No. 2005/0254009, incorporated herein by reference, areintegrated; the motorized chin rest responds to User Interface inputfrom the Iris Viewer viewport. User Interface input provided byselecting a point in the Iris Viewer (e.g. mouse click) instructs themotorized chin rest to move the patient in the X-Y plane so that theentry point of the scan beam 110 corresponds to the point selected inthe image. In order to focus the iris image, user input commands theUser Interface CPU to instruct the motorized chin rest to move thepatient along the Z-axis. Once the iris and pupil are in focus, thecorrect working distance between the instrument and the patient's eye isset. Construction of the instrument is such that the scan beam pivotpoint is approximately at the focal plane of the iris viewer. Thus,setting the correct working distance puts the pivot point of the scanbeam in the plane of the patient's iris. It would also be possible tohave a system wherein the housing that contains the optical elementmoves and the patient is stationary. In that case, the alignment wouldbe performed by moving the housing.

The second step in proper patient alignment for this system is aligningthe optics to correct for refractive error. In FIG. 1, the lower leftviewport 200 displays a summary image 205, nominally a real time fundusimage from a fundus camera or a line-scanning ophthalmoscope (LSO) orother fundus imager. The system of U.S. patent Ser. No. 11/820,773achieves its best correction for refractive error when the retinal imageof the fundus imager is optimally focused. In order to focus the retinalimage, the user provides input commands to the User Interface CPU toinstruct the motorized chin rest to move the patient and ocular lens incombination along the Z-axis. In this case, the ocular lens is alsomoved so that the distance between the ocular lens and the patientremain fixed, thereby retaining the Iris viewer focus and preserving thepivot point alignment.

The third step in proper patient alignment is aligning the OCT imager toimage at the correct depth. In FIG. 1, the lower left viewport 200displays a summary image 205, nominally a real time fundus image or anintegrated OCT en-face image. During cube scan alignment, the system ofU.S. patent Ser. No. 11/820,773 displays line segment overlays 210, 230,and dashed white segments not seen below the lines 240 and 260 in thesummary image outlining the extent of the acquisition volume (the volumeto be acquired). Scan location icons 220, 240, 250, and 260 overlay thefundus image in the summary window indicating the location of fouralignment B-scans. In viewport 300, images 320 and 350 show scans fromlocations 220 and 250, respectively. These are the central vertical andhorizontal B-scans, respectively. Images 340 and 360 are reduced sizeimages of the bottom and top B-scans of the acquisition volume, takenfrom locations 240 and 260, respectively. The user provides inputcommands to the User Interface CPU to instruct the OCT imager to set theOCT image delay line so that the retinal image in displays 320, 340,350, and 360 is optimally located.

Optionally, a fundus image or integrated OCT en-face image from aprevious examination can be overlaid on the live fundus image 205 in asemi-transparent manner. Aligning the previous OCT en-face image and thelive fundus image ensures that the volume within the region bounded by210, 230, 240, and 260 will image the same region as was acquired duringthe previous exam. When a previous fundus image is registered to the OCTen-face from the same exam, aligning the previous fundus image with thelive fundus image enables the acquisition of the same (or nearly thesame) OCT volume region as was acquired during the previous exam. Thisoptional step is preferentially performed after refractive erroralignment and before completion of the step setting the OCT imager delayline for systems using a spectral domain OCT imager.

The term en-face appears in a variety of forms in the literature.Various authors use at least three forms: en face, en face, and en-face.All forms are equivalent and, in the field of ophthalmology, an OCTen-face image is an image extracted from a 3D OCT volume by integratingthe OCT signal along a viewpoint, generally over a range of depths, asdescribed in Knighton, et al., 20060119858.

It is important to control the position of the patient's eye. Eyemovement causes the OCT imager to view different regions of the eye.Moving the eye can be useful when performing the optional alignment to aprevious exam described above. Moving the eye can also be used toachieve a particular imaging path (to avoid a particular part of thecornea, which might be damaged, or to avoid a cataract in thecrystalline lens.) The User Interface (UI) provides access to control animage fixation subsystem. The patient fixates on a target projected bythe fixation subsystem. Through the fixation subsystem, the UI controlsthe position of the eye by controlling the location of the fixationtarget image. The line-scanning ophthalmoscope (LSO) fundus imager andfixation subsystem described in U.S. patent Ser. No. 11/820,773 areintegrated; the fixation target 270 of the fixation subsystem respondsto User Interface input from the summary image (LSO) viewport. DuringOCT image capture, the patient focuses on a fixation target, helping toreduce or prevent eye movement. The UI allows the user to select a pointin the summary image (e.g. mouse click) and the UI CPU instructs thefixation subsystem to move the fixation target in the X-Y plane so thatas long as the patient follows the fixation target, the selected pointbecomes the center of the fundus image. Thus, the center of theacquisition volume (in the X-Y projection plane) becomes the selectedpoint in the fundus image.

During OCT image depth alignment, Viewport 300 of FIG. 1 displays fouralignment B-scans of the acquisition volume. Through these four images,the user can verify proper patient alignment. Clearly, those versed inthe art will understand that other images can be used for this purpose.For example, images obtained from locations 210 and 230 can replaceimages 340 and 360, obtained from locations 240 and 260. Alternatively,two B-scans diagonally intersecting the image cube are theoreticallysufficient, without any additional border images. However, the imagesshown in Viewport 300 are sufficient and readily understood.

The description provided here for optimally aligning OCT depth range forOCT volume capture describes OCT volume imaging of the fovea. Oneskilled in the art can readily generalize to imaging the optic nerve orother imaging applications (including the imaging the cornea). Foracquiring volume cube scans using the UI of FIG. 1, after the scanningbeam entry point is aligned (steps 1 and 2 above), proper OCT depthrange alignment can be attained by locating five (5) points: theextremal point of the retina within the cube scan and the four points ofthe retina at the intersections of the planes bordering the cube scan(four corner points). The extremal point of the retina is the retinalpoint furthest from the imaging device. The four corner points of theretina are the four points of the retina closest to the imaging devicealong the four lines formed at the intersections of planes bounding thesides of the cube. The central point of the retina is the retinal pointat the center of the X-Y projection of the volume. For proper alignment,the extremal point should be at or near the central point and the fourcorner points of the retina should be within the scan volume. The UI ofthis invention displays regions of the volume cube in neighborhoods ofthe 5 points and accepts user input to direct moving the 5 points towhere they should be. Aligning these 5 points to their optimal locationsis sufficient to align the OCT depth for proper patient imaging.

Images 320 and 350 are the central vertical and horizontal B-scans ofthe acquisition volume, respectively. The retinal curvature within theseimages is determined by the degree of myopia of the subject eye. Forfoveal imaging, the scan beam entry point is properly located when theshape of the retina in images 320 and 350 is approximately symmetricabout the fovea and the fovea is located approximately one-half wayacross each image. If the retina is not symmetric about the fovea, theuser adjusts the entry location 110 through the Iris Viewer viewportinterface. If the fovea is located approximately one-half way acrosseach image, the fovea is properly centered in the X-Y plane. If thefovea is not properly centered in the X-Y plane, the user adjusts thecenter of the acquisition volume within the summary viewport. The entryangle centered on the fovea is adjusted by moving the fixation target torelocate the eye in combination with moving the entry point to re-centerthe fovea.

The volume alignment process ensures that the tissue of interest will bewithin the volume scanned. The UI displays B-scans 350, 320, 340, and360 so that, if the retina is within the depth range of each of theseB-scans, it is with high probability within the depth range of theentire volume. (Mathematically, for a retina with smooth anterior andposterior surfaces and without inflection or saddle points, the retinawill be within the volume range with probability 1.) We first ensurethat the point of the retinal image that is furthest from the imagingdevice will be within the volume scanned. After the previous alignmentsteps are properly performed, this point will lie near the intersectionof the line normal to the central horizontal scan 350 through the pointof the retinal image that is furthest from the imaging device in scan350 and the line normal to the central vertical scan 320 through thepoint of the retinal image that is furthest from the imaging device inscan 320. When the imaging system is properly aligned, this point is on(or nearly on) the line segment at the intersection of scans 320 and350. Because of the continuity and curvature of the retina, the extremalpoint is within the volume scan if the extrema of the retinal arcswithin scans 320 and 350 are each within their respective images (orsufficiently within their respective images when compared to thecurvature of the retina and their offset from the extremal point.)Placement of the extremal point within the image volume ensures that theimage of the retina does not “drop out” of the bottom of the volumeimage.

The user ensures that the retinal image does not “pop through” the topof the volume cube by checking images 340 and 360. If the retinal imagelies within each edge (A-line) where the sides of the cube meet, thenthe corner points of the retina lie within acquisition volume and theretinal image will be within the volume cube. Thus, for every horizontaland vertical slice of the volume, the retinal image will remain withinthe slice. That is, if the retinal image is visible within each of theedges of the cube, then the OCT depth range is correctly set and the OCTimage cube range is properly aligned. The advantage of a User Interfacedisplaying images as in viewport 300 is that a user viewing the fourimages can quickly and easily determine if the acquisition volume isaligned for retinal image acquisition. That is, if the upper and loweredges of the retina are visible across all four images displayed and theextremal point is visible within the images, then the acquisition volumeis aligned for capturing the retinal volume image. In other words, ifthe Retinal Nerve Fiber Layer (RNFL) and Retinal Pigment Epithelium(RPE) are visible across all four images displayed, the acquisitionvolume is aligned.

In SD-OCT systems that have not otherwise eliminated the mirror image inthe spectral detection path, one needs to ensure that it is the image ofthe retina and not the mirror image of the retina that is visible withinthe four edge A-lines. For ease of use and consistency in the display,two tomograms from opposite sides of the volume cube are displayed,rather than simply the four edge A-lines of a cube scan. Theoretically,the four edge A-lines (the first and last lines of the two tomogramsfrom opposite sides of the cube) contain enough information to determinethe appropriate SD-OCT depth range. However, displaying the twotomograms from opposite sides of the cube simplifies both the displayand the user's ability to understand the situation. In FIG. 1, images340 and 360 enable the user to ensure that the retina will be within theacquisition volume. Additionally, any mirror image will appear folded inthe tomogram. Image folding at the top of the slice informs the userthat the OCT range is set too deep and that they should adjust the OCTengine to image more shallowly. These tomograms contain the informationof the A-lines, which is displayed at the ends of the tomograms.Alternatively, tomograms from locations identified by 210 and 230 couldbe used instead of the tomograms 340 and 360 because the informationcontained in the four A-lines needed is also contained within thosetomograms. Thus, the User Interface notifies the user if image foldinghas occurred and enables them to adjust the OCT range to correct thealignment.

Since display space is limited, one UI goal is to minimize the number ofimages needed for alignment. However, since patient alignment andimaging is the ultimate goal, different displays that make alignmenteasier, whether by making training easier, by making volume manipulationeasier, or by any other means, can be added to the final UI arrangement.The UI of record displays the critical image locations and provide ameans to relocate the acquired image volume to properly position thecritical image locations within the volume.

Proper alignment requires the user to associate information containedwithin the various images in each of the three viewports of FIG. 1. TheUI simplifies this association by providing slice locators to identifythe location within the fundus image of Viewport 200 of the B-scanspresented in Viewport 300. The color (yellow) of ID icon 355 and slicelocator 250 shows the correspondence between B-scan 350 of Viewport 300and its location (indicated by line segment 250) within the fundus imageof Viewport 200. The ID icon 355 can contain additional sliceinformation, such as the direction of the scan. In this instance, thehorizontal yellow bar in the icon identifies image 350 as a horizontalB-scan. Slice 320 is the vertical B-scan from location 220. The color(white) of ID icon 325 and slice locator 220 shows the correspondencebetween B-scan 320 and its location (indicated by line segment 220.) Thecorrespondence between slices 340 and 360 and their locations in thefundus image (indicated by segments 240 and 260) is indicated by thecolor (blue) of ID Icon 345 matching the color of slice locator 240 andthe color (magenta) of ID Icon 365 matching the color of slice locator260. The UI uses color in slice locators and ID Icons to simplify userassociation of images to location. Alternatively or in addition, the UImay use the same color to highlight the border of the slice display.

Finally, the User Interface provides a manual control (not shown inFIG. 1) for polarization compensation between reference and source armsin the OCT imager. Theoretically, this control would control the threepolarization paddles necessary for complete compensation of polarizationdifferences between the reference and source arms. In the systemsdisclosed in U.S. patent Ser. No. 11/820,773 a single polarizationpaddle is used to simplify the interface and approximately compensatefor polarization differences. In this design, a single slider, knob orsimilar interface is used to move the paddle while the user views theimage, looking for the position of the control that maximizes the signalcontent of the retinal image.

In FIG. 1, the fundus image 205 is an LSO image. The real time fundusimage can be from any fundus imager, such as a fundus camera, a scanninglaser ophthalmoscope (SLO), or a line scanning laser ophthalmoscope(LSLO), or a line-scanning ophthalmoscope (LSO). Any confocal fundusimage is advantageous over any non-confocal fundus imager, like a funduscamera, since the confocal image eliminates or reduces glare andbackground information away from the focal plane, creates sharplydefined images and can be simultaneously acquired with the OCT volumescan when separate wavelengths are used. Confocal imaging producesimproved vessel imaging over traditional fundus cameras. Simultaneousimaging is preferred because of shortened exam duration and highercorrelation between images. LSO images are acquired even faster than SLOimages because of the simultaneous imaging of a line. LSO imagingdiffers from LSLO imaging largely in that the laser of an LSLO imager isreplaced by a

In the system described in U.S. patent Ser. No. 11/820,773, OCT volumescans are commonly called cube scans. However, not all edges arenecessarily the same length. In fact, the opposite sides are notnecessarily parallel, nor are the top and bottom necessarily flat, sothese volumes are not, strictly speaking, even cuboid. The volume maymore properly be called a regular 4-sided truncated spherical pyramid.Regardless, the term “cube” is generally used to indicate this nearlycuboid volume with nearly parallel sides and almost planar top andbottom. Clearly, other volumes would suffice as well, such as nearlyright regular n-gons or nearly regular truncated n-sided pyramids.

Automatic Patient Alignment

Optionally, prior to volume acquisition, the OCT system canautomatically align the retina within the volume scan. To accomplishthis, the User Interface CPU causes the system to acquire a limitednumber of B-scans, performs image processing on the B-scans, determinesthe location where the retina would appear within the volume scan if thevolume scan were performed under this configuration, and re-aligns thesystem for proper retinal imaging, if needed, before acquiring thevolume scan. The alignment steps are the same as for manual alignment.The system first aligns the subject's pupil with the scan beam. That is,the system sets the proper working distance and initial entry point. Thepatient sits and rests her head in the motorized head support apparatus.The Iris Viewer captures an image, like 120 in FIG. 1, and passes it tothe host CPU, which identifies the pupil or iris. The Iris Viewerrepeatedly captures iris images, passing them to the host CPU. The CPUdetermines from the images where the initial X-Y alignment should be andthen commands the motorized head support to position the patient for thescan beam to pass through computed point on the pupil. Initial X-Yalignment can be determined by applying standard image processingtechniques, such as edge detection. For example, the pupil edge can bedetected by thresholding the image of iris image to find the pupilboundary. The CPU also estimates the working distance (the distancebetween the ocular lens and the pupil or iris) based on the sharpness offocus of the iris and/or pupil. The CPU commands the motorized headsupport to move the patient's head in the Z direction to set the properworking distance. This is a well-known auto-focus problem. One means offocusing is to move the patient's head repeatedly, analyzing iris imagesat multiple locations until an optimal focus is achieved. In alternatesystem designs, the working distance can be set by moving the ocularlens instead of the patient. For the imaging system of U.S. patent Ser.No. 11/820,773, this working distance alignment sets the pivot point ofthe scan beam substantially in the plane of the patient's iris.

After the working distance is set, the system automatically aligns theoptics to correct for refractive error. In one instantiation, the systemwill automatically acquire a retinal fundus image using a fundus imagersuch as a Line Scanning Ophthalmoscope (LSO). The host CPU processes theimage data to determine sharpness of focus of the retinal image. Again,this poses a well-known auto-focus problem. Alternatively, the systemmay automatically acquire a B-scan using the OCT imager. The host CPUprocesses the B-scan image data to determine relative strength of theOCT signal. The host CPU commands the motorized head support to move thepatient's head and the system's ocular lens module in combination in theZ direction to focus the fundus imaging system to accommodate therefractive error of the eye. The LSO and OCT systems are designed sothat, when the LSO fundus image of the retina is in focus, the OCTimager will produce a sharp retinal image in each B-scan. Because theLSO and OCT systems are co-aligned, the optics correction for refractiveerror can be performed automatically using either B-scan signal strengthor fundus image sharpness. The refractive error correction adjustmentpreserves the pivot point alignment by moving the head and system ocularmodule as a unit. At this point, the working distance and refractiveerror correction are set. For improved focus, the refractive errorcorrection step may be repeated after setting the OCT depth range asdescribed below.

The final positional alignment is automatically set to align the OCTimager to image at the correct depth. One means to accomplish OCT rangealignment is for the system to acquire central horizontal and verticalB-scans like 350 and 320 displayed in FIG. 1. (Central horizontal andvertical B-scans are B-scans essentially slicing across center of theOCT volume, either horizontally or vertically, respectively.) Forretinal imaging of the fovea, the host CPU can process these images tolocate the fovea. Segmentation of the retinal image identifies theretinal fovea by searching for the foveal pit. The fovea can also beidentified by pattern recognition techniques, or by other imageprocessing techniques. If the segmentation indicates that the RetinalPigment Epithelium (RPE) near the fovea lies below the volumeacquisition region, then the host CPU commands the OCT depth delay toaccommodate imaging at a deeper depth. The host CPU determines theoffset needed to center the foveal pit within a volume so that it iscentered in the central horizontal and vertical B-scans. The host CPUthen commands the Fixation Target subsystem to move the fixation targetso that the patient's eye is redirected so that in the new acquisitionvolume, the central horizontal and vertical B-scans have the foveal pitcentrally located. The foveal pit has been identified and centered usingonly a few B-scans and without a full volume acquisition.

The host CPU commands the Fixation Target subsystem directly. Thepatient fixates on the fixation target. The fixation target is afiducial mark generated at a visible wavelength. The fixation target isfocused into the eye at a location calculated to cause the patient toorient their eye in a specified direction. When the host CPU moves thefixation target, the patient rotates their eye to follow the fixationtarget. The pupil rotates with the eye, changing the center location ofthe pupil. The host CPU commands the motorized chinrest to move sidewaysto compensate for pupil motion. Thus, the host CPU determines the offsetneeded, computes the fixation target location needed to sufficientlycause the eye to move to achieve this offset, causes the fixation targetto move with the eye following and adjusts the chinrest so that theentry point within the pupil remains optimal.

The host CPU adjusted the OCT range alignment so that the retinal imagedoes not extrude through the bottom of the volume; i.e., the point onthe Retinal Pigment Epithelium (RPE) furthest from the imaging device iswithin the volume scan. The point on the RPE furthest from the imagingdevice should be directly below the foveal pit in the most recentlyacquired central horizontal and vertical B-scans. This is the extremalpoint of the RPE. If the extremal point of the RPE does not intersectthe bottom of either B-scan, the margin available to keep the extremalpoint in the volume is stored. Setting the OCT range so that theextremal point is just above the volume floor ensures that as much ofthe retina as possible will be within the imaging volume. However,because of edge effects, placing the preferred imaging region centrallywithin the image volume is advantageous. For this reason, it is alsouseful to find the retinal points within the volume to be imaged thatare closest to the imager.

In order to determine if these retinal points closest to the imager arewithin the volume to be acquired, two additional B-scans along theboundary of the acquisition volume are acquired. These two scans containthe edges of the acquisition volume, preferably B-scans from oppositesides of the cube like 340 and 360 in FIG. 1. The upper boundary of theretina, the inner limiting membrane (ILM), is segmented in these images.If the ILM extrudes through the top of either B-scan, measurements areperformed on the segmented retinal images to estimate the extent towhich the retinal image extrudes through the top of the volume. Thisestimate can be easily performed using a parabolic fit to the ILM withinthe B-scan. Data indicating the available margin for top adjustment arecompared with data indicating the available margin for depth adjustmentand a final adjustment is made. The system is automatically aligned forvolume acquisition and the volume acquired. In this way, automatic depthadjustment can compensate for variations in the length of the human eyewithout user intervention. Optionally, once the CPU determines thelocation of the fovea, it can command the motorized head support to movethe patient in the XY plane, moving the entrance location of the scanbeam in the patient's pupil to partially level the appearance of theretina in B-scans 320 and 350. This is most useful when the ILM or RPEdo not appear symmetric within the limited alignment B-scans.

System volume scan alignment can be automatically optimized even if itis not possible to adjust the system so that the retinal image is fullywithin the volume scan. The system automatically optimizes scanalignment based on a priori defined imaging preferences. For example,centering the retina within the volume is one optimization criteria.Maintaining image quality of the retinal point furthest from the imagingdevice is another. In general, the system can automatically adjust imagealignment of any tissue of interest, provided the tissue can beidentified and a metric applied to the measure of how well the tissue isaligned.

SD-OCT systems that have not otherwise eliminated the mirror image inthe spectral detection path can automatically choose the portion of theimage that moves consistently with the depth adjustment and reject themirror image that moves in the opposite direction to the depthadjustment. Because of this, automatic depth adjustment can be used todiscriminate between the SD-OCT image and its mirror. Alternatively,chromatic dispersion mismatch can be used separately or in conjunctionwith depth adjustment to determine which half of the SD-OCT output isthe image and which half is the reflection. While the image and themirror image have the same integrated intensity (when integrated overlinear intensities), the true image is sharper and has greater peakintensities. Any measure of the center of the image that weights higherintensity points more than linearly in intensity, will be biased towardthe true image. The centroid along depth Z of the square of theintensity in the B-scan is one example measure of the depth position ofthe retina.

Automatic polarization compensation between reference and source arms inthe OCT imager may optionally be automatically performed in conjunctionwith or after aligning the optics for refractive error correction. Inthe systems with a single polarization paddle, a simple techniquecomprised of setting the paddle and examining the resultant image signalcontent can rapidly scan through multiple paddle settings to maximizethe signal content of the retinal image.

Image Acquisition

Once the imaging system is aligned, the OCT volume is acquired. Duringacquisition, the User Interface removes the alignment overlay andreplaces it with a live display of the OCT fundus image. An OCT fundus(or en-face) image is an image created from OCT data by integrating overdepth. For performance advantageous, dedicated hardware or firmware inthe data acquisition path computes the live display of the OCT fundus,accumulating the signal across depth cells of the A-line as the A-lineis acquired. Using this dedicated hardware approach, the B-scan imageand the associated line of the OCT en-face image are available fordisplay at the same time, with minimal delay following opticalillumination. The live B-scan can replace any one of the alignmentB-scan images, preferably in the largest image display window available.The User Interface displays live B-scan in one Viewport and OCT en-faceoverlaying the LSO fundus display in another Viewport provides the userwith real-time information for quality control of the OCT volumeacquisition. This UI also provides the user with rapid feedback on thestatus of the volume acquisition. Since it is common for the user to askthe patient to hold still during the volume capture, it is reassuringwhen the user has a visual queue showing the exam completion status. Theuser can assess imaging artifacts in real-time and can initiatere-acquisition quickly and easily with a single command. On completionsof volume acquisition, the UI automatically displays a movie of the OCTacquisition B-scans. The speed of the movie playback is variable.Alternatively, a playback mode where the B-scans are displayed quicklyfor the first and last scans captured and more slowly for the centralB-scans allows the user to quickly skim the edge volume information andmore carefully examine the B-scans near the center of the volume. Asingle command can initiate image volume archival.

During acquisition, it is advantageous to acquire not only the OCTvolume, but also, for a portion of the volume, to acquire one or morehigh-definition B-scans. High-definition B-scans are scans of higherresolution than other B-scans within the volume, either by reducing theA-lines spacing within a B-scan or by reducing the B-scan spacing withinthe volume. Time and data storage constraints limit the number ofhigh-definition scans within the volume. It is advantageous for thecentral horizontal and vertical B-scans of the volume acquisition to behigh-resolution.

Real time high frame rate imaging of the fundus enables the operator toobserve in real-time the relative position of the scan area to thefundus. This real-time capability allows the operator to position thescan pattern/area over the area of interest even in the presence offrequent eye movements.

Maximum Intensity Projection

One analysis application or tool for analyzing OCT volumes is theMaximum Intensity Projection (MIP). MIP is a volume rendering techniqueused to extract high intensity structure from volume data. Live OCT MIPdisplays provide the user with another tool for viewing the quality ofthe OCT volume acquisition. Once the imaging region is aligned andvolume acquisition begins, the User Interface replaces the alignmentdisplays with a live display of the OCT en-face and one or more MaximumIntensity Projection (MIP) display. For the standard MIP display, at anytime during acquisition, the current MIP scan is a pixel-by-pixelmaximum of the previously acquired B-scans. That is, for the firstB-scan of the volume scan, the MIP scan is the same as the B-scan.Thereafter, when the system acquires the N+1^(st) B-scan of the volume,the N+1^(st) MIP scan is the pixel-by-pixel maximum of the pixel valuein the N+1^(st) B-scan and the pixel value of the N-th MIP scan.Mathematically,

M_(ij) ^(N+1)=max(M_(ij) ^(N),B_(ij) ^(N+1))

where B_(ij) ^(N) is the value of the pixel at coordinates (i,j) of theN-th B-scan and M_(ij) ^(N) is the value of the pixel at coordinates(i,j) of the N-th Maximum Intensity Projection scan. Initially, M_(ij)¹=B_(ij) ¹. The MIP display clearly indicates retinal movement out ofthe acquisition volume. This display provides a rapid quality check ofthe acquired volume for most eye motion, providing the user with earlynotification of one of the most common causes of volume scan failure.

In general, an MIP is a projection of parallel rays through a 3D volumeonto a plane perpendicular to the rays. The viewpoint is the directionof the parallel rays. The value at a point in the projection plane isthe maximum of the values of the 3D volume along the path of the raythat intersects the projection plane at that point. Generating MIPsalong a plurality of viewpoints generally improves volume visualization.MIPs generated along a plurality of viewpoints forming a simple curvecreate the illusion of volume rotation when played back sequentially.For these purposes, the MIP need not be computed in real-time.Background computing and processing of an MIP may identify abnormalitiesand the UI can automatically notify the operator.

Other intensity projections are known and useful. The most common threereal-time MIP displays use viewpoints along the X-, Y-, or Z-axis. Theexplanation above described the MIP with viewpoint along the Y-axis.This MIP displays an apparent B-scan slowly changing over time.Horizontal or vertical integration of each B-scan forms the MIP alongthe X- or Z-axis. Thus, for each B-scan of the volume acquisition,projection forms a single line for each of these MIPs. Alternatively, aMinimum Intensity Projection (MinIP) may be formed to extractlow-intensity structures from the volume data. Currently, specular noiseproduces enough dark regions within tissue to adversely impact MinIP ismany applications. However, MinIP is useful for viewing trulynon-reflective regions and the scope of its usefulness improves as imagespeckle is reduced. Those versed in the art will readily see otherfunctions that can be applied to OCT volume data along a viewpointuseful for rendering intensity projections of other features within thevolume.

Image Analysis

MIP analysis is an example of an image analysis application. Analysisapplications perform image analysis on acquired data, and the analysisis available to the user through the User Interface. In some instances,applications automatically select relevant images and analysis fordisplay. Some viewports display single images while others displaysequences of images (“cine” or movies). Optionally, simultaneouslydisplayed movies are registered and synchronized. When applicable,display locators overlay one view of the object locating the regiondisplayed in another view.

By design, the analysis portion of the User Interface efficientlydisplays relevant images to the user and provides access to imageanalysis tools. For analysis, the UI displays a combination of imagesincluding fundus, en-face, processed OCT and OCT images. For OCT imageanalysis, the UI displays one or more high definition scans along with acollection of sub resolution images, called thumbnails, each of which isassociated with a high definition scan or slice. Viewports present OCTimages either in full resolution or as partial resolution thumbnails.Registered images can be displayed stand-alone, side-by-side oroverlaid. Overlays can be displayed in color or black-and-white and withvarying degrees of transparency. Overlays on high definition displaysmay need to be upsampled to achieve the same sampling density, whileoverlays need to be downsampled to overlay thumbnails.

The UI supports both image acquisition and analysis. Image data isacquired in scan patterns using a system scan sequence. In most cases,scan patterns are designed so that data is acquired along lines(B-scans) or collections of lines, such as a rasterized volume. Volumescans can be cube, starburst, spiral, or other collections of lines thatfill a volume at some resolution. A typical volume image acquisition isa collection of imaging planes that fill the volume.

In acquisition mode, the UI supports both alignment and review. In FIG.2, the UI displays the system scan sequence identifier (or scan type) inthe fundus viewport 200′. In FIG. 2, the raster scan icon 285 identifiesthe scan type as a raster sequence. The scan type can be displayed astext or icon anywhere in the display area or be available through apop-up or pull down, but it is advantageous to display a scan type iconor thumbnail, overlaid on top of one of the image displays so that it isalways available but minimally disturbs the image content. However, theuser is able to toggle the icon, to hide it so that the underlying imageis fully visible. It is advantageous to display one or more scans infull resolution 380 while other scans appear as thumbnails. “Fullresolution” here is resolution relative to the display, not the dataitself. Displaying the image in full resolution means presenting it inthe highest resolution available for this display, window, or viewport.The image data may well have more lines than the CRT has pixels, butpresenting the image in the highest resolution window available istermed here to be full resolution. It is also advantageous that thedisplayed full resolution scan be selectable by interaction with thescan type icon. For example, selecting element 290 from the raster icon285 causes scan 380 to be reduced in size to a thumbnail while scan 390is displayed in full resolution and no longer reduced in size. Whetheror not selectable from the icon, the scan is preferably identifiedwithin the scan type icon 285 by color-coding the relevant element 280in the scan type icon 285 with the color used in the ID icon 385.Alternatively or in addition, as noted above, the UI may use the samecolor to highlight the border of the slice display.

Slice Locators

FIG. 3 depicts an example User Interface for analysis. This UI displaysa summary image, in this case an LSO image, and transverse slices of theOCT volume, with locator indicia to help locate the slice of interest.An LSO image 400 is located in the upper left viewport of the UI,providing a summary overview of the eye with anatomical landmarks.Overlays of analysis images on the summary image clarify the anatomyassociated with the analysis. The user can access any one of a number ofdifferent types of overlays through a drop down menu 410. Commonanalysis overlays are thickness maps, confidence maps, and OCT en-faceimages. If the user does not choose an overlay, the UI defaults tooverlaying slice locators and an outline of the volume acquisitionregion on the LSO image. A transparency control 420 allows variablelevels of transparency in the overlay. Here transparency means aweighted blending of the overlaying and underlying images. Variabletransparency assists the user in clarifying the anatomical location inone extreme and viewing the analysis image in the other.

The UI enables the user to view volume slices individually as stillimages or collectively as sequences of slices presumptively calledmovies. A movie play button 430 activates playing the sequence of slicesfrom the current Active Plane. The Active Plane can be horizontal,vertical, or depth. In FIG. 3 shows the horizontal plane in viewport441, the vertical plane in viewport 451, and the depth plane in viewport461. Horizontal and vertical slice locators are indicia, indicating therelative location of respective volume slices. The horizontal slicelocator 440 and the vertical slice locator 450 identify the location ofthe horizontal slice 442 and the vertical slice 452. The scan ID icon445 uses color to identify the association between the slice locator 440and the slice 442. The scan ID icon 445 further includes graphicinformation showing that this is a vertical scan. Additionally, theslice border 448 is the same color as the slice locator 440, wherein thecolor identifies the correspondence between the location in the fundusimage and the displayed volume slice. Scan ID icon 445 and slice border448 are each indicia within the horizontal display viewport indicatingthat volume slice 442 corresponds to the location indicated by slicelocator 440. A play sequence button 443 is associated with thehorizontal slices in viewport 441. Activating the play sequence buttonwithin the horizontal plane window causes the horizontal plane to becomethe Active Plane and plays the horizontal slices in sequence, like amovie. As the slices play through the movie, the slice locators in thevertical and depth windows (e.g., slice locator 464 in depth window 461)update the location of the displayed horizontal slice. Similarly, thehorizontal slice number is also updated with the movie.

The vertical slice identified by slice locator 450 is located in thelower left viewport 451, here the vertical plane viewport. The scan IDicon 455 again uses color to identify the association between the slicelocator 450 and the slice 452. Additionally, the slice border 458 is thesame color as the slice locator 450. Clearly, the association betweenthe volume slice 452 and location 450 in the fundus image does notrequire both indicia 451 and 455. While each presents its own in ease ofuse value and both may be present, either one establishes thecorrespondence between image and location. Play sequence button 453 isassociated with the vertical slices in viewport 451. Activating the playsequence button within the vertical plane window causes the verticalplane to become the Active Plane and plays the vertical slices insequence, like a movie. As the slices play through the movie, the slicelocators in the horizontal and depth windows (e.g., slice locator 465 indepth window 461) update the location of the displayed vertical slice.Similarly, the vertical slice number is also updated with the movie.

In FIG. 3, depth slices are presented in viewport 461, making 461 thedepth plane viewport. A yellow slice locator 460 and the slice number(19) 459 within viewport 451 identify the location of the depth C-scanslice displayed in image 462. Also, a yellow slice locator 466 and theslice number (19) 449 within viewport 441 identify the location of thesame depth C-scan slice displayed in image 462. A C-scan is a slicetaken from the volume at a fixed depth. The slice border 468 is the samecolor as the slice locator 460, wherein the color identifies thecorrespondence between the slice locators in the horizontal and verticalimages and the displayed volume slice. Slice locator lines 464 and 465indicate the location of the horizontal and vertical scans,respectively. Activating the play sequence button within the depth planewindow causes the depth plane to become the Active Plane and plays thedepth slices in sequence, like a movie. Just as for horizontal andvertical movies, as the slices play through the depth movie, the slicelocators in the horizontal and vertical windows (e.g., slice locator 466in horizontal display window 441 and slice locator 460 in verticaldisplay window 451) update the location of the displayed depth slice.Also, the depth slice number is updated in sequence with the movie.

Selecting a slice locator in any window, typically by moving the mousepointer over it and clicking, activates it, selecting also the ActivePlane. Manipulating the position of the active slice locator, typicallyby click and drag, causes the corresponding slice to be displayed in thewindow associated with the Active Plane and updates all other views.That is, when slice locator 466 is selected, the depth plane is selectedas the Active Plane. Dragging slice locator 466 to a new slice updatesthe slice number 449 and image 462 in viewport 461. At the same time,slice locator 460 is updated to the new depth slice location and thedepth slice number 459 is updated. Alternatively, entering a slicenumber (another indicium) selects an Active plane, causes the slicelocator to move to a new location, and the causes the UI to display thecorresponding slice from that location in the window associated with theActive Plane and update all other views.

For improved viewing of the underlying image, the UI enables the user totoggle the display of slice locators as well as other overlays.

In FIG. 4, we see an example of the analysis interface images for astarburst pattern. An LSO image 500 is in the upper left of theviewport. The starburst scan type icon 510 overlays the LSO image withslice locators showing the locations of the starburst B-scans of the OCTvolume. All six slices of this starburst pattern appear as thumbnails,with the B-scan displayed in thumbnail 523 from location 513 alsoappearing as a full resolution image 530 on the right. The B-scandisplay includes an icon 525 associating the displayed image 530 withthe corresponding location from which the B-scan was acquired. Icon 525is a possibly decimated copy of the icon 510 displayed in the LSOwindow. Because icon 525 may be too small to observe the scan locatorwithin the icon easily, it is advantageous to display a border 533 aboutB-scan 530 that is the same color as is the border of the selectedB-scan's displayed thumbnail 523.

Volume Scrolling

The UI provides a number of features to enable users to scroll throughvolume data. FIG. 5 shows an LSO image with horizontal 440 and vertical450 slice locator indicia. Elsewhere, images extracted from thelocations marked are displayed. The color of the slice locator matchesthe color of the border of the matching slice's border within itsviewport (not shown here). The dashed lines 401, 402, 403, and 404indicate the boundary of the acquired volume. The identified volumeslice number is displayed beside horizontal slices (as illustrated by490) or below vertical slices. Users can update the slice locators (andthe corresponding displayed slices) by clicking within the LSO image(the intersection of the slice locators moves to the click point), or byclicking and dragging one of the slice locators. In addition, the ActivePlane Indicator 495, the colored arrow to the left or above a slicelocator, defines which slice locator is “active”. Users activate a planeby clicking on its slice locator or active plane indicator. Users canthen scroll through the active plane using the mouse's scroll wheel (notshown) or by playing the sequence of scans in a movie by activating themovie button (not shown). Placing a fovea marker 470 over the locationof the fovea may help orient the user.

Movie Mode

When simultaneously displaying a summary view and a sequence of B-scansor slices as a movie, the UI synchronizes the movie with the slicelocator in the summary view so that the correct slice locator displayedin the summary view corresponds to the volume slice displayed in themovie. When the UI plays two or more movies of volumes acquired for thesame patient during separate visits, their volumes can be registered andsynchronized before playing. Generally, users prefer to view the moviessimultaneously, where side-by-side displays show similar regions ofanatomy. However, for some cases, it is easier to see anatomicaldifference when the UI presents the movies sequentially, in particular,sequentially interleaved.

Volume registration can be global, regional, or local. A globalregistration of the volumes provides the best single co-ordinatetransformation associating the two volumes, but may be inappropriate ifthere is eye movement in one or both volume images. When artifacts arepresent in one or both volume images, such as those caused by eyemovement, it is often better to identify and remove the artifacts beforeregistering the volumes. Alternatively, if the artifacts are identified,but not removed, the volumes can be registered by separately registeringregions within the volumes where there are no (or limited) artifacts.Alternatively, deformable registration of the volumes may account formotion artifacts.

The movies from separate visits are registered and synchronized. Whenplayed, synchronized movie frames display volume slices showingcorresponding regions of anatomy. Because motion artifacts can creatediscontinuities in the volume data, it is often best to perform volumeregistration region-by-region or even slice-by-slice, rather than havinga single registration offset for the entire volume. A movie may beplayed back in slow motion or at high speed. That is, the UI providesfor variable rate movie playback. The UI also provides for variableplayback frame rate between scans. In one instance, the movie is sloweddown when displaying slices from the central region of the volume(generally the region most interesting to the user) and played fasterfor the first and last slices of the slice sequence (when fewer detailsof interest are generally observed.)

LSO CINE—Integration of Signal Between Depths

Multiple fundus images are acquired during the exam. Since the LSO scanrate is faster than the OCT volume scan rate, several LSO scans areacquired during an OCT volume scan. For some exams, it is advantageousto combine LSO images before display in order to achieve improved imagequality. Noise in the LSO images can be reduced by temporally averagingthe images. This smoothing can be performed over disjoint sequences ofLSO images, with the accompanying reduced sampling rate. However, thesampling rate can be kept constant, either using an IIR filer or byusing an FIR filter with a time-late display. The simplest such FIRfilter is the boxcar filter, where a fixed number of LSO images areaveraged. Still other combinations of LSO images can be used to improvethe LSO image display.

Measurement Tools

The UI provides access to measurement tools. The simplest measurementtool is the distance measurement tool, which measures the pixels orvoxels between two specified points in an image and converts thismeasurement to distance units. Other measurement tools are perimetertools, area tools and volume tools. The Thickness Measurement tool is aparticularly important distance measurement tool. FIG. 6 illustrates athickness map.

Thickness Measurement

Users access the Thickness Measurement tool through the UI in analysismode. The Thickness Measurement tool, shown in part in FIG. 6, allowsthe user to determine the retinal thickness at any point on the retinawithin the OCT volume. By moving the pointer 610 over the thickness mapimage within the Thickness Measurement tool (accessible from the Toolbarin the UI in analysis mode), the user determines the location to analyzeand the Thickness Measurement tool determines the distance from thelocation to the center of the fovea and the retinal thickness at thelocation. When a mouse-over pointer is detected on any 3D map, a line615 is projected from the pointer tip perpendicular to and intersectingthe surface of the map. This line and surface intersection point aredisplayed simultaneously on all maps (although only the intersection isshown on 2D maps such as those overlaid on LSO images). The thickness atthe point located is displayed 620 in a popup text string below eachmap. The popup string 620 displays the thickness at 610 and the distancefrom the point 610 to the center of the fovea marker 625. These valuesupdate as the user moves their mouse over the map surface. When themouse leaves the map, the thickness and distance measurements disappear.

The Thickness Measurement tool measures retinal thickness from oneretinal layer to another. Various authors and investigators usedifferent retinal layers to determine retinal thickness. The ThicknessMeasurement tool allows for different retinal thickness measurementsusing a drop down menu 630 from which the user can choose from a varietyof thickness measurement definitions. The choice displayed in FIG. 6 isto measure thickness from the ILM layer to the RPE layer. Alternatively,one might choose to measure a portion of the retina, such as the nervefiber layer thickness. Other choices for measuring the thickness withinthe retina may be included in this interface.

Patient Information Area

The UI also provides access to the Patient Information Area. The PatientInformation Area is a page containing basic information, including suchitems as: Patient Name, Patient ID, Gender, DOB, Ethnicity, Doctor,Acuity, HIPAA information, exam specific information such as eyeexamined, reason for exam and diagnostic information, and other patientspecific information. The exam study archive includes PatientInformation. The system also stores re-exam specific information for thespecific patient in the patient exam archive. Re-exam information isinformation such as the headrest configuration and alignment settingsneeded to reposition the patient automatically during a second or latervisit. For instance, the system records the location and orientation ofthe headrest components for the exam. This includes the setting of thevertical and lateral position of the chin cup, the tilt angle of theheadrest mechanism, the refractive error correction to focus the retina,the depth range setting to center the B-scan, and the polarizationcompensator setting. Thus for each patient the system stores thealignment parameters necessary to re-align the patient at a returnvisit. For example, the saved parameters could be: the distance fromforehead to chin, the depth from the forehead to the vertex of the eye,the horizontal distance from the centerline of the head to the eye, therefractive error, eye length, and corneal birefringence (the dominanteffect on polarization setting). These system settings can be restoredon a second or later visit, saving time in system set-up and alignment.

Exam Archives

The patient exam archives may be subsequently retrieved for furtheranalysis, for comparison to another exam, or for some other reason. TheUI provides an interface to search the patient exam archives andretrieve archived exams and studies. Studies are multiple exams linkedtogether by a user because of common characteristics such as patient ordisease. FIG. 7 illustrates an exam imaging system capable of archivingexams. Patient exam archives can be on the local machine 710, on anothermachine networked 715 to the current machine. The networked machine canbe another imaging system, a central exam storage server or any othermachine capable of archiving data and accessing a network. The UIimplements exam archive management functions such as: exam retrieval,database synchronization, archival of exam studies, archive searches andother data archive management functions. The UI can implement userauthorization checking before performing archive functions. The user canperform searches based on patient information, doctor information, examinformation, diagnosis, or other relevant information. The UI includes asummary page where for rapid review of current and archived exams invarious formats. The UI provides exam summaries in text fields (name,date, etc.) and/or by image thumbnails. Archived information includespatient set up information. Using patient set up information, thepatient is quickly repositioned on subsequent visits (as discussedabove).

In response to an action by the user, the UI will interact with theprocessor 720 to find all visits (or some portion of all visits)satisfying some search criteria. For example, using a single action, theuser may request all exams performed on a specific patient and the UIwill display a registered image from each visit on screen 725. Factorsdetermining which exams are available include the type of analysisperformed, the availability of data, the operator's access privilege,and limitations in local storage.

Thumbnail

UI display space is limited. Exam thumbnails can be extremely helpful insummarizing a particular exam and finding it within a large collectionof exams. In some cases, a search using patient information, doctorinformation, or diagnosis is quite successful. In other cases, it isuseful to see some typical image data from the exam or study. Imagethumbnails summarizing the exam simplify exam retrieval. Imagethumbnails are stored with the exam and displayed in the exam retrievalUI. Image thumbnails may be automatically chosen, such as a retinalthickness map, or the user may identify one or more images thatspecifically identify the exam and the UI will make thumbnails of themfor exam identification. Users can readily ascertain exam details byviewing one or more of the exam thumbnails. Hyperlinks or other activelinks associate thumbnail images with the originating exam or study.Executing the link of an exam thumbnail retrieves the desired exam ordisplays the desired view. For example, a thumbnail image could be amaximum intensity projection along the fast scan axis, along the slowscan axis, or along the depth scan axis. Alternatively, the thumbnailmight be a fundus image, OCT en-face image, a thickness map image, or aB-scan image.

Composite Thumbnail

One available thumbnail summary is the composite thumbnail shown in FIG.8. The composite thumbnail is a summary indicator that combines, in athumbnail, various components of the actual data set in a reducedresolution format. A composite thumbnail is a thumbnail composed of acollection of thumbnails. A composite thumbnail is often composed ofthumbnails of different display types or formats, such as B-scans, OCTen-face, fundus, or analysis maps. FIG. 8 is a cartoon of a collectionof composite thumbnails displayed on a screen. FIG. 8 shows one actualcomposite thumbnail, with placeholders for many on a page. Eachcomposite thumbnail is associated with an exam or an exam analysis anddouble clicking on the composite thumbnail can retrieve the associatedexam or analysis. The composite thumbnail shown in FIG. 8 combines anLSO image 640 with thickness map overlay 645. It also includes slicelocators 641 and 643 indicating the location of 2 tomograms. Alsoincluded in the composite thumbnail is an OCT en-face image 649 overlaidby a scan type icon 647, in this case indicating a raster scan. The twothumbnails 631 and 633 on the right of the composite thumbnail arethumbnails of the two tomograms indicated by 641 and 643, respectively.Display resolution and real estate limit the size and number ofcomposite thumbnails presented. The simplest composite thumbnail, notcounting a simple thumbnail, is a decimated image with overlay, such asan LSO with thickness map overlay or an OCT en-face with exam typeindicator overlay.

Control Elements

For user convenience, when space allows, the UI has some controlelements embedded in the viewport. For example, the Iris viewport hasavailable space on most display devices since the Iris display itself issquare and most display devices are wider than they are tall. As shownin FIG. 9, the chinrest controls 750, focus control 752, and button 755providing access to other controls are located in the Iris viewport nearthe Iris display. Button 755 provides access to additional controls,such as brightness and contrast controls. Alternate controls, includinghard key controls, can replace soft controls displayed in viewports. Forexample, the chinrest and focus controls, may be hard key controls orsoft key controls accessed via a viewport without display images. The UIdefault disables the Iris reset button 758 until an adjustment is made.While other implementations are possible, the location of the controlsis easy to find and natural for most users, simplifying user training.In another instantiation of this interface, the user accesses thecontrols through menus or pop-up windows accessed by clicking on acontrol location. In yet other instantiations, the user accesses thecontrols through menus or pop-up windows available through contextsensitive cursor sensitive graphics areas, where the action of theinterface device is dependent upon the content of the display area.

FIG. 10 is a display of the summary (LSO) image Viewport with UIcontrols embedded in the display. A user adjusts the horizontal andvertical volume acquisition region by means of a 2-D motion button 760while slider 763 controls the focus. Button 765 provides access to othercontrols such as brightness and contrast. The LSO reset button 768 isnominally disabled until an adjustment is made. Embedding UI controls inthe analysis UI viewport simplifies user access and training. In otherinstantiations of this interface, the UI controls do not require realestate in the analysis UI, e.g., UI controls are accessed through apop-up or drop-down by clicking on a control button or activated byplacing the cursor in a context sensitive area activating the controlpop-up window or drop down menu.

Image Registration

FIG. 11 is a flow diagram illustrating a method of registering twoimages. These images are from the same patient 800, of the same view,and the same eye; but are acquired at different times. One image of acurrent exam may be retrieved 810 from either memory or local hard driveor both images 810 and 810′ may be retrieved from the local hard driveor network storage. Image registration gives us a common coordinatesystem between the two images. After registration, when we look atcorresponding locations in each image, we know that we are also lookingat the same physical location in the patient's retina.

Images of the same eye generally have the same underlying structure,such as the retinal vasculature, which is consistent over time.Underlying structure in each image is detected in 816 and 816′, andthese underlying structures are matched and aligned to each other 818.Registering the underlying structure of one image to the underlyingstructure of the other registers the images to one another.

The underlying structure we are interested in is the vasculature of theeye. First, we smooth 812 and 812′ the speckle while preserving theedges of the underlying structure. There are many mechanisms forsmoothing speckle such as filtering, using boxcar filters, with theinherent image artifacts, or smoothing with other low pass filters, likeGaussian filters, with somewhat fewer artifacts. Edge blurring isinherent in these methods. High pass filters can enhance edges, creatingtheir own inherent image artifacts. Well known methods, such as thosedeveloped by Sobel, Canny, Haralick or others (see Pal, N. R. et al.,Pattern Recognition, Vol. 26, No. 9, 1277-1294) , can be used to detectedges. After edge detection, resolution of vessel interior can beproblematic in target rich environments with large numbers of vessels,especially if the vessel sizes vary. Time permitting, techniques likePerona and Malik's anisotropic diffusion (see Perona, P., Malik, J.,IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 12, No. 7,629-639) or Yu's speckle reducing anisotropic diffusion (see Yu, Y.,Acton, S., IEEE Trans. On Image Processing, Vol. 11, No. 11, 1260-1270)provide improved performance by smoothing interior regions whilepreserving edges. Indeed, any filter capable of smoothing speckle noisefrom optical imaging data without blurring edges is likely to findapplication here, especially anisotropic filters, with independentsmoothing and edge preservation in different directions.

Smoothing the image without significant blurring of edge boundariesprovides some enhancement of underlying structure. Further enhancementis needed for some images. Smoothing without edge retention reducesvessel wall contrast, reducing the detectability of the underlyingstructure. Depending on the underlying structure and background noiselevels, gamma correction may be sufficient for enhancement. However, ingeneral, a structure enhancement 814 and 814′ process is more robust. Ithas been demonstrated that the eigenvalues of the Hessian of the imagedata can be used to enhance tubular regions within an image. Theeigenvalues of the Hessian derived from regularized derivatives areknown to be more stable. Regularized derivatives are derived byconvolving the image with the derivatives of a Gaussian kernel ofappropriate scale chosen by choosing the σ of the Gaussian. Frangi (inFrangi, A., et al., IEEE Trans on Medical Imaging, Vol. 18, No. 10,946-956) and Sato (in Sato, Y., et al, Medical Image Analysis,2(2):143-168) have demonstrated this technique for enhancing tubularregions, with emphasis on enhancement, and binarization, respectively.

Once the underlying structure is determined in both images, theseunderlying structures are represented as images 816 and 816′ that areregistered to each other. Typical registration techniques use rigidregistration to register equally scaled images. Elastic registrationtechniques register images that are not equally scaled. Underlyingstructure images are registered either using a single process or stagedusing a coarse-fine registration technique. Coarse-fine registrationobtains the final registration result in stages. In the first stage,decimated images are registered. Registering decimated images reducesthe computational complexity by reducing the registration search space.This is the coarse registration. Fine registration then registers thehigh-resolution images. The fine registration also has a reduced searchspace, because the coarse registration sets a starting point and boundsthe search extent. Correlation, sum squared difference, sum absolutedifference, Bayesian maximum likelihood, and/or cost function metricscan be applied to create a measure for choosing the best-matchedregistration. Once the underlying structures are registered, theoriginal images are themselves registered to each other.

The resulting registration can be displayed using different colorchannels for each image of the registered pair. The vessel enhancedbinary or grayscale images can be converted from intensities of white tointensities of a unique color (preferably a color channel color)allowing their overlay to be more easily visualized. In this fuseddisplay, vessels that are properly registered overlay perfectly and taketheir color from both color sources (or channels), forming a new color,while vessels that are not properly registered do not overlay each otherand appear in the combined image in their original color. In thisdisplay, the quality of the registration is immediately apparent to theviewer.

Intelligent Data Review

There is a distinct need in the field to display relevant information inan efficient manner. FIG. 12 illustrates a user interface displayingrelevant information in an efficient manner. A summary image 640′ (inthis case an LSO fundus image) is registered to a thickness map 600,which is overlaid 645′ over the fundus image 640′. The thickness map isitself registered to the volume. A region of interest (ROI), in thiscase the fovea, is identified within the LSO image (or the thicknessmap); and the appropriate horizontal and vertical slices of the volumeare located (641′ and 643′), extracted from the volume and displayed631′ and 633′, respectively, showing the region of interest within thevolume. As shown in FIG. 12, the summary image data 640′ and volume data(not shown) do not have to be acquired by the same imaging device or bythe same subsystem within an imaging device, but they do need torepresent the same region. Because the summary image is registered tothe volume, the relevant slice of the volume associated with the ROI inthe summary image can be extracted from the volume and displayed. Notonly is the relevant slice of the volume readily available to the user,but also the thickness map overlaid over the fundus image efficientlydisplays the metric (thickness) within the context (fundus image).

Also shown in FIG. 12 are images showing the upper and lower surfacesfrom which the thickness map is computed. In this case, image 600 is thethickness from the internal limiting membrane (ILM) to the retinalpigment epithelium (RPE). Item 607 shows the ILM over the RPE. Since theRPE is mostly hidden in this image, item 605 clarifies the RPE image byshowing the RPE surface alone.

As described above, the identified ROI within the image need not bedirectly registered to the volume. Rather, the identified ROI within theimage may be indirectly registered to the volume through one or moreintervening registrations. For example, an OCT volume and an en-faceimage derived from it are inherently registered. FIG. 13 illustrated anen-face image overlaid over an LSO summary image. In this example, inorder to improve visualization of the overlay, the en-face image isslightly misaligned and transparent, so that the difference in theimages is more readily visible. An en-face image 649′ and an LSO image400′ are registered. Registration may be accomplished through commonimage device coordinates, through image processing image registrationtechniques or by manual manipulation. Identification of a region ofinterest like the fovea within the LSO image identifies one or moreslices through the fovea within the volume. In many cases, thehorizontal B-scan of the OCT volume through the fovea is the preferredslice because it is the slice acquired requiring the least time foracquisition. The horizontal B-scan is the volume slice least likely tobe compromised with motion artifacts. In other imaging systems withdifferent scan sequences, this minimal time slice through the fovea maybe vertical rather than horizontal.

FIG. 14 illustrates another efficient presentation of relevantinformation, such as automatic identification and display of slicesassociated with a lesion. Previously, a sequence of slices extracteduniformly from OCT volumes displayed regions suspected of containingretinal lesions. Nominally, the sequence displayed was a sequentialsequence of B-scans. By automatically selecting relevant slices fordisplay, we display information more efficiently by showing more datathat are relevant in fewer images.

First, we display a summary scan 850. While this summary scan can be anLSO image, it is usually more relevant to display, or at least overlaythe display, with a thickness map. Lesions are associated with peaks inthe retinal thickness map. As shown above, the derivation of the retinalthickness map registers it to the OCT volume data. Identification of apeak in the thickness map 852 identifies one or more slices 854 of thevolume through the peak. While the UI enables the user to choose thepeak or peaks in the thickness map manually, there is a strong advantageto having one or more peaks automatically detected. Any one of manypeak-picking methods can identify the most relevant peak. While thetallest peak is readily ascertainable, in many instances, the peaksupported by the largest volume is of equal or greater importance. Thetallest peak, if unsupported by neighboring thickness in the thicknessmap, may be merely noise.

Particular care must be taken when locating more than one peak. Onemethod for finding secondary extrema is to choose appropriatelyconstrained local extrema, for example, peaks separated by a minimaldistance. Another method utilizes a contour map derived from thethickness map. After choosing the first peak, a threshold is setdefining a neighborhood about the first peak wherein a second peakcannot reside. One such threshold requires a valley of at least aminimal depth between two peaks. Alternatively, the valley depth mightbe a function of the peak heights. Another such threshold requires aminimal separation between peaks.

Once the application locates peaks, it extracts and displays 856 one ormore slices of the OCT volume through the peaks. Extracted slicesshowing peaks in thickness are more relevant than equally spaced slices.Nominally, the application chooses B-scans for display, since a B-scanis the volume slice least likely to be compromised with motionartifacts. However, slices chosen to illustrate other features of thelesion, such as breadth or volume may be chosen for more efficientanalysis of the lesion.

The format of an efficient image display depends on the anatomy imaged,the analysis conducted, and the content of the images. When the userselects an analysis tool, a display format is selected to display datarelevant to the anatomy and analysis. Algorithms within an analysis toolapplication are designed to detect features or anomalies, or enhanceidentifiers associated with a specific pathology. For example, inretinal analysis, the thickness map enhances lesion detection andenables automatic lesion detection. Large lesions extend across multipleB-scans. The application identifies the lesion correctly in multipleneighboring B-scans. However, it is redundant to display the samepathology repeatedly. For this reason, tools are designed to segment thevolume into specific regions of interest and extract metrics that areindicative of the features of the specified pathology relative to thespecific region under examination. Slices representative of the variousregions are displayed, or slices specific to a region generallyindicative of the extent of the pathology within the region are chosenfor display. Display space is limited. A limited number of slices,images, or maps are presented in the UI, usually between two and five,with additional information available using overlays. The UI displaysmore items when limited resolution is sufficient. When items do not needto be simultaneously available, the UI scrolls one or more images offthe viewing area.

Report

Selected images are automatically integrated into a report. The reportcan be subsequently reviewed, printed, or electronically archived.Automatically selected images can be reviewed, accepted, and/or replacedby manually chosen images selected by the user. Once the report isaccepted, it can be archived or printed with a single command.

Automatic Identification of Suspicious Results

There is a distinct need for automatic identification of problematicdata and suspicious analysis results. Herein disclosed is a system forautomatically identifying and displaying a suspicious segmentationresult for review and/or correction. A measure of confidence in asegmentation result is established, estimating the probability ofsegmentation errors and providing a mechanism to select segmentationresults with low confidence for user review. This measure of confidenceis called a confidence map. The number, location and confidence ofsuspect results are determined and incorporated within the confidencemap. Some or all of the suspect segmentation results can be displayedfor user modification. When automatic propagation of segmentationmodifications is enabled, segmentation corrections are propagated withinthe slice and into nearby slices. Thus, enabling automatic propagationreduces the number of suspect segmentations displayed to the userbecause only one slice in a cluster of suspect slices needs to bedisplayed for possible correction.

The confidence map may be generated either during or after thesegmentation procedure. In one embodiment, specified steps and portionsof the segmentation process are associated with elements of a costfunction used to develop the confidence map. Examples of elementsassociated with the confidence map are: image intensity, localvariations in intensity, measure of continuity (or discontinuity) ofsegmentation results, measures of variation in segmentation depth andother measures of internal segmentation consistency, strength of imagegradients, the number of detectable layers or edges, feature shape andorientation, and geometric proximity (say to boundaries of the imagedarea and/or segmentation results and other measures of a prioriinformation about the segmentation and the surrounding environment).Both theoretical and heuristic factors are included to improve theprecision of the confidence map.

A confidence map is a record of the degree of certainty of thesegmentation operation at each point of the segmentation. In oneembodiment, the confidence map is a weighted function of confidencemetrics. Each confidence metric estimates the degree of certainty of aparticular aspect of the segmentation at each point. Confidence metricscan be developed around any of the elements associated with theconfidence map. For example, one metric is the intensity of the signalat the segmentation boundary. This metric is generally weighted moreheavily for segmentations based on signal strength, such as RPE boundarysegmentation, but is not weighted as heavily for segmentations based onimage gradients, such as the ILM boundary segmentation. Another metricis the strength of the image gradient at the segmentation boundary.Metrics can be developed for each of the elements mentioned above, aswell as for other features used or useful for segmentation decisions. Ingeneral, since the physical layers being imaged and segmented areexpected to be unbroken, abrupt discontinuities in the segmentation areweighted low in confidence. The greater the discontinuity, the lower thecontinuity confidence metric is valued. Confidence metrics may be basedon the segmentation alone or any combination of segmentation and imagedata. When more than one feature is being segmented, a confidence metricmay be based on any combination of data from one or more segmentationresults and image data. For example, when both the RPE and ILM aresegmented, a confidence metric combining the two segmentations is theconsistency of the segmentations of the RPE and ILM. Another combinationmetric is the continuity of the measurement of the distance between theRPE and the ILM.

The confidence map is a combination of the individual confidencemetrics. For example, when each confidence metric is non-negative, thesum of the individual metrics, or the sum of the squares of theindividual metrics, or a normalized, weighted sum of the individualmetrics are all instances of confidence maps. For example, oneconfidence map is a normalized weighted sum of four confidence metrics;m_(I) (for intensity), m_(D) (for discontinuity), m_(C) (forconsistency), and m_(G) (for geometry):

½{½m_(I)+·m_(D)+½m_(C)+·m_(G)}

Similarly minima, maxima, medians, products, weighted products, andnormalized weighted products of the individual metrics also provideinstances of confidence maps. An example of a weighted product ofindividual measures m₁ and m₂ is m₁·√m₂ and a normalized weightedproduct is (m₁·√m₂)^(2/3).

Alternatively, the confidence map may be derived directly from one ormore segmentation results either including or without including directcomputations on image data. It is not necessary that the confidence mapbe computed from individual confidence metrics. For example, aconfidence map may be the output of a properly trained neural net. Theneural net can be trained to recognize segmentation errors from thesegmentation results, image data, and a set of training data withsegmentation errors identified by an expert.

Comparison of the segmentation and the original image providesadditional factors for the confidence metric. Reasoning combinesindividual measures into the confidence metric by formulas, logic,partial information logic (fuzzy logic) or even neural networks. Manyconfidence metrics are equivalent and each is dependent upon thethreshold at which decision points are set.

When displayed, the confidence map can overlay the thickness map, theLSO or other fundus image, or the confidence map display maystand-alone. The confidence map can be used to modulate a thickness map,another confidence map, or any display of the segmentation. For example,a normalized confidence map can be used for transparency control of athickness map overlay. In areas of complete confidence, the thicknessmap is completely opaque, while in areas of no confidence, the thicknessmap is completely transparent. This overlay provides the user with avisual representation of the thickness map where the segmentationconfidence and a transparent view of the fundus where the segmentationconfidence is low. Another combination example is multiplying thenormalized confidence map of the RPE segmentation by the normalizedconfidence map of the segmentation of the ILM. This is one example of aconfidence map of the thickness estimate. Combinations need not onlyinclude combinations with confidence maps. One such combinationmodulates an en-face image by a thickness map. A new image is formedusing the intensity of the en-face image while adding color, where thehue of the new image is proportional to the thickness in the thicknessmap. Of course, any fundus image can display retinal thickness using anycolor map by means of a look-up table or other function associatingthickness with color.

Summarizing each line of the 2-D confidence map by a statistic of thatline projects the 2-D confidence map into a 1-D confidence line. Astatistic searching for lines with low confidence could use the minimumvalue statistic. A mean or median statistic provides an estimate of theaverage confidence of the segmentation in the line. Such a measureprovides an estimate of the confidence that takes significantly lessdisplay space. Displaying a confidence line using the minimum statisticalong the side of an overlay like the thickness map overlay provides animmediate indication of which slices are likely candidates forsegmentation errors. Displaying a confidence line using one statisticalong one edge of an image and another confidence line using a differentstatistic along the opposite edge provides additional information. Onesuch display appends the confidence line derived using the meanstatistic along one edge and the confidence line derived from thestandard deviation statistic along the opposite edge, providing the userwith an estimate of the average confidence of the segmentation within aB-scan and the variance of the confidence within the B-scan with minimalimpact on the thickness map display itself.

Intelligent Boundary Editing: Edit-Propagation

Nominally, a volume is composed of a collection of B-scans. Onesegmentation methodology segments each B-scan separately, associating asegmentation confidence with each segmented point of the B-scan. Just asthe individual segmentations can be joined to provide a segmentationmap, the resulting confidences can be arranged in a confidence map. Asuspect point of a confidence map is a point where the confidence maphas a low confidence value in a region of interest. A suspect region ofa confidence map is a region where the confidence map has low confidencevalues throughout. When a confidence region is suspect, the segmentedimage and the suspect segmentation of that image are displayed. It isuseful to display both the image and the segmentation in a singleviewport, with the segmentation overlaid on the image. For improvedvisualization of the image, the segmentation overlay can be transparent.The flow diagram of FIG. 15 illustrates this process. If thesegmentation appears visually incorrect, the user enables a segmentationeditor and modifies 880 the segmentation for that image. When automaticpropagation of segmentation modifications is enabled, the automaticpropagation application automatically propagates the modificationthrough neighboring image segmentations. It is not necessary to updatethe confidence map to include the high confidence of the segmentationmodification. If the confidence map is updated, the confidence of amanual edit is set very high, ostensibly set to 1, where 1 is thehighest possible confidence.

When sufficient information is available within the image, the automaticpropagation application recomputes the neighboring segmentation resultsstarting from the known good result provided by the user. Alternatively,when insufficient information is available within the image to create ahigh confidence segmentation even with the known good result provided bythe user, the propagation application propagates the user modifiedsegmentation by interpolation 882 between the user modified region andthe region of high confidence. It is also possible to combineinterpolation techniques and recomputing segmentation techniques byconstraining the segmentation technique to its best result within aneighborhood of the interpolated result. High confidence regions neednot be large segments. Interpolation between high confidence regions assmall as individual points is useful in extending segmentationboundaries. If there are no local regions of high confidence, the usermodified data points can still propagate the modification by smoothingthe modification to neighboring segmentation results. Alternatively, thesystem can display additional regions of low confidence for usermodification.

Once the segmentation modification within an image is complete 884,edits are automatically propagated across neighboring images 886.Allowing the local modification to initialize a definite boundary,automatic modification propagation can use the segmentation algorithm toextend the segmentation, using the user modification as a startingpoint. Alternatively, interpolating between the known good segments canfill a small gap between two high confidence segments of a knowncontinuous object within an image. Extrapolation from high confidencesegments can also be used to extend the segmentation boundary.Extrapolation from known good points can also extend segmentationresults into regions of low confidence.

When making corrections to the segmentation of a three-dimensionalvolume of image data, it is helpful to propagate those correctionsautomatically to nearby regions that require similar corrections. Inthis way, segmentation corrections require limited, if any, repetitiveediting. After the user modification of one or more segmentation points,the automatic propagation module changes neighboring segmentationresults in a manner consistent with the volume image contours and in amanner also consistent with the segmentation.

In one embodiment, the extent of the automatic edit propagation (thepropagation region) is proportional to the distance of the correction.In this case, for greater correction distances, the automaticmodification process will extend its update of segmentation resultsfurther away from the edited point. In another embodiment, the extent ofthe edit propagation is proportional to the distance to the nearestknown valid segmentation result. This segmentation result may be knownto be valid because it was, itself, a user edit, or it may be known tobe valid because the confidence map at this point exceeds a fixedthreshold. In an embodiment wherein the confidence map is recomputed forupdated points, the edit propagation may extend until the newly computedconfidence at a point fails to meet a minimum confidence level. Any ofthese or combinations of these methods can be used to determine theextent that the edit is propagated. These methods need not be symmetricand will normally depend on the distance between samples in theextension direction.

In one embodiment, an analysis comparing the edited points to theneighboring segmentation results identifies the propagation region. As afirst pass, automatic propagation interpolates provisional segmentationresults within the propagation region using the edited points and theprior segmentation at the boundaries of the propagation region.Automatic propagation may refine the provisional results, producing newsegmentation results that more closely follow the contours found in theimage data.

For example, the user is presented a summary image display such as anOCT en-face image, a thickness map, or a 3-D rendering of the layersegmentation. The user selects a B-scan from the image volume and theB-scan is displayed, including the segmentation results overlaying theB-scan image within the selected image display. The user selects one ormore segmentation locations within the display, identifyingmodifications to the segmentation. (If there is only one segmentationlayer within the display being reviewed, there is no ambiguity regardingwhich segmentation is being modified.) If more than one segmentationresult is displayed, the user can manually select which segmentationresult is being modified, or the segmentation result being modified canbe automatically selected by the system, for example, by choosing thesegmentation closest the edited point or points.)

In one embodiment, the propagation region is a fixed M×N area where N isthe number of pixels along the fast scan direction and M is the numberof pixels in the slow scan direction. In another embodiment, the area ofthe propagation region depends on the size of the modification. In yetanother embodiment, the propagation region extends to the nearest highconfidence segmentation result in every direction from the modifiedpoint. Propagation regions are areas enclosed by discontinuities of thesegmentation results. Progressively lowering the threshold used todefine a discontinuity establishes a threshold that in turn identifies aregion or set of regions containing all of the edited points. The binaryimage defined by this threshold can be refined by morphologicaloperations, defining a preliminary propagation region. After somerefinement by morphological or other processing methods, this binaryimage defines a region for edit propagation. This region is modified toaccount for previous user modification that should not be altered, orother specific constraints on propagation, if needed. After accountingfor such constraints, the edit propagation region is defined.

Automatic propagation interpolates across the propagation region fromedited points to the segmentation surface at the periphery of thepropagation region. When the propagation region extends to the edge ofthe image, automatic propagation extrapolates from the edited point(s)to the image edge in a manner reasonably consistent with both thenearest edited point and the nearby edges of the propagation region.This creates an initial correction to the segmentation over thepropagation region.

In the next step, examining the data in the vicinity of the surfacerefines the interpolated surface. This vicinity may be defined using thedifferential threshold used in defining the propagation region, by theelevation of the nearest edited points, and/or the segmentation surfaceelevations along the periphery of the propagation region. Searching inthe vicinity of the interpolation, a strong edge (maximum significantaxial gradient) in the image typically refines the ILM would, while themaximum significant intensity typically determines the RPE. In the eventthat there is no significant image information, as in the case ofshadows or broad regions without distinct gradients, the interpolationis used for the segmentation correction.

To complete the process, automatic propagation applies post-processingto the segmentation corrections. In its simplest form, automaticpropagation smoothes the segmentation result to remove any edges. Insome instances, automatic propagation applies a more complex nonlinearprocess such as high order spline interpolation or median filtering,within the propagation region and possibly extending to the periphery.

An automated editor for propagation of edits is also capable of makingimperfect edits. The system includes a capability to review the results888 of the propagated edits. If another segmentation requiresmodification 890, that slice and segmentation are displayed 892 and theediting process continues until all modifications are complete 894. Aninfinitely alternating “limit cycle” of correcting corrections isavoided by ensuring that the manual edits themselves should not bealtered by a subsequent automatic propagation of changes.

Choice of Overlay

Analysis results on 2-D and 3-D datasets are often displayed as images,which, for medical imaging, have improved value when they are registeredto the anatomy. While the analysis itself is registered to thecoordinate system of the data analyzed, the results need to beregistered to the anatomical location that they represent in ameaningful way for the user. For most analyses, users have the option ofoverlaying analysis results on the LSO fundus image. Thickness maps,confidence maps, en-face images, binary images and other images withuseful diagnostic information can be derived from the OCT volume data,which is registered to the LSO fundus image. Displaying the analysisresults over the LSO image provides context in which to interpret theresults. Users can choose the analysis results to overlay and set thetransparency of the overlay. Transparency in the overlay enables clearervisualization of the underlying LSO image and better context for theanalysis results. Variable transparency allows the user to show more orless detail in the analysis overlay. The ability to toggle the overlayallows the user to view the anatomy and associated analysis in rapidsuccession.

On a second or later visit, analysis images from previous exams areavailable and can be overlaid over the current summary image. Anyprevious LSO, OCT en-face, or analysis image of this patient can beregistered with the current exam and can be used to overlay a currentsummary display. Overlays archived with the exam are available forvarious applications such as registration and image retrieval. Forexample, an OCT image from a previous exam overlaid on the current LSOimage during pre-acquisition provides a visual indication of a possiblemisalignment and can be used to help realign the optics to the sameorientation as used in the previous exam.

Progression Analysis

It is advantageous for medical practitioners to assess changes in tissueover time. The presently described User Interface displays changes inthe anatomy of the tissue over multiple examinations. Viewing changeover time is useful for monitoring, the progression of a disease orpathology, or the response of tissue to therapy. FIG. 16 shows a flowdiagram of a process for displaying registered images from differentexams. Initially the system is enabled to acquire data 870. A previousstudy is retrieved 872 from archive for comparison. If a summary image(OCT en-face or LSO or other fundus-like image) of the previouslyacquired exam is available, it can be overlaid over the current summaryimage as described in co-pending U.S. patent application Ser. No.11/717,263, filed Mar. 13, 2007, publication 2007/0216909, which ishereby incorporated by reference. This enables the operator to positiona new scan over the previously scanned area with high degree ofaccuracy. Alternatively, the system can register the current en-faceimage to the previous en-face image and automatically position a newscan over the previously scanned area, also described in co-pending U.S.patent application Ser. No. 11/717,263. Once the alignment is complete,the volume acquisition begins. During acquisition, the alignment overlayis replaced by a live display of the OCT fundus image, enabling qualitycontrol of the OCT volume scan during volume acquisition. The images tobe compared are selected 874 and registering the display images 876minimizes the remaining differences in acquisition coordinates oranomalies. Corresponding regions of display images from each exam aresimultaneously displayed to visualize change 878. As noted above, theside-by-side movie is especially useful for comparing changes inpathologies from visit to visit.

Typically, the first exam performed is the baseline. However, the usercan choose any exam in the patient archive to be the baseline exam.Images and image analysis from more recent exams are compared to thebaseline exam. When more images are available for comparison than fit onone viewing screen, the additional images are available through a scrollbar or through another image-paging tool. The UI allows the user toreorder images so that the user can compare images in close physicalproximity. The system retains the original order so that images can beredisplayed in chronological order, when needed.

The primary change analysis display, illustrated in FIG. 17, provides aseparate row for each exam. On the left side of the row is a fundusimage, 910, 910′, 910″, with one or more tomogram locators. In thecenter of the row one or more tomograms, 920, 920′, 920″, 930, 930′,930″, are displayed. These are the tomograms indicated by the locatorson the fundus image. Different rows display different exams. In order toidentify change, the display shows the same areas of tissue in thedisplays from the different exams. The image data from the differentexams are registered, either to data from a single exam, such as thebaseline exam, or through a series of connected registrations, such asregistering each volume image to the volume image from the previousexam. In this way, multiple volumes are registered either directly orindirectly to other volumes from different exams. The registered imagesfrom different exams can be simultaneously displayed. The registrationcan be done using summary data, such as LSO fundus images registered toen-face images, through direct volume registration, by registeringvolume regions, or by registering individual B-scans. The UI enables theuser to scroll through a selected volume. Synchronization is enabled sothat scrolling through one volume scrolls through multiple volumessimultaneously, with registered images from different exams displayedsimultaneously. Movies are enabled so that playing the movie for oneexam is synchronized with movies playing in another exam.

Other mechanisms for displaying change are also available. Whenselected, change statistics are displayed. For example, in FIG. 17,maximum thickness change plot is displayed in viewport 940. Ofparticular interest is any change in the thickness or volumemeasurements of one or more of the various intra-retinal regions betweeneye examinations. Users can choose the intra-retinal region of interest(ILM to Bruch's, ILM to RPE, RPE to Bruch's, Region Threshold,User-Drawn Region), and the statistic used to evaluate change (MaximumThickness, Average Thickness, Center Thickness, Volume). Displays fromthe region of interest using the chosen statistic are displayed for themultiple visits.

Additional displays are available, such as thickness maps, differencemaps, and pathology maps. FIG. 17 displays two pathology map overlaysover fundus images, 950 and 950′. The pathology map is an overlay withopaque regions of pathology and transparent normal regions. Choosing anadditional display either adds it to the row or replaces a display inthe row with it. A difference map between the thickness map of thebaseline and the thickness map acquired at a later visit could replacethe pathology map displayed in FIG. 17, or be added in a new column, inthis case requiring the user to scroll in order that it be visible. Athickness map replacing a fundus image retains the locator featurereferencing the location of the slices and the synchronization featuresynchronizing the locator with the slices being displayed. Movies remainenabled when the thickness map replaces the fundus image. The variousimages retain their common co-ordinate system derived throughregistration of the volume images.

While the description herein describes macular change analysis, theinvention is equally applicable to change analysis utilizing image datain other fields, such as glaucoma image change analysis, cataract imagechange analysis, retinitis or retinopathy image change analysis, andother image change analysis of disease related to the eye or othertissue for which an imaging modality provides a tool for analysis.

Progression analysis measures change over time by monitoring anattribute. The relevance of that change is determined by comparing themeasurements either to baseline data or to a model.

Better Measurements of the RNFL

Volume scans make it possible to improve measurements of the RetinalNerve Fiber Layer (RNFL). A common method of viewing RNFL thicknessmeasurements is to measure the thickness of the RNFL in a cylindricaltomogram centered on the optic disk and plotting the resulting thicknessmeasurements. Misplacement of the cylindrical tomogram creates anomaliesin the resulting plot. Since the RNFL is normally thinner further fromthe optic disk, if the cylinder is displaced from centering on the opticdisk, the region of the cylinder further from the optic disk normallymeasures the nerve fiber layer thinner while the region of the cylindercloser to the optic disk normally measures the nerve fiber layerthicker. Given a 3D volume image of the optic nerve head, one can make ameasurement of the RNFL thickness that does not depend on an arbitrarymeasurement cylinder. The optic disk can be identified within the volumeand the proper misalignment avoided. Also, additional data from thevolume can be used to statistically improve the measurement data.Additionally, circle scans of different radii can be extracted from thevolume data. A study (see Carpineto et al., European Journal ofOphthalmology, vol. 15, no. 3, 2005) has shown that the mean thicknessvariability is drastically reduced when the size of the ONH was takeninto account.

In the cylindrical tomogram, the thickness of the RNFL is determined bysegmenting the line at the top of the nerve fiber layer and the linecorresponding to the boundary of the retinal pigment epithelium (RPE).For each point on the RPE boundary, there is a closest point to the topof the nerve fiber layer, and the distance to this closest point is ameasure of nerve fiber layer thickness. The set of such thicknessmeasures can be plotted as a function of position on the RPE boundary.In the volume scan, the thickness of the RNFL is determined bysegmenting the top surface of the nerve fiber layer and the curvecorresponding to the boundary of the retinal pigment epithelium (RPE).For each point on the RPE boundary, there is a closest point to the topsurface of the nerve fiber layer, and the distance to this closest pointis a measure of nerve fiber layer thickness. The set of such thicknessmeasures can be plotted as a function of position on the RPE boundary.The display of the thickness map is a surface. That is, the proposed newmeasure of RNFL finds the edge of the hole in the RPE where the opticnerve exits the eye. This edge is a curve. The distance from a point onthis curve to the segmentation of the top of the RNFL is computed. Theaverage of these distances is a measure of the RNFL thickness in theneighborhood of the ONH. Alternatively, a plot of these distances aroundthe ONH shows the relative thickness about the ONH.

This specification describes various instantiations for efficientlyproviding relevant image displays to the user. These displays are usedto align patients, locate display images within other display images,automatically display suspicious analysis, automatically displaydiagnostic data, simultaneously display similar data from multiplevisits, improve access to archived data, and other improvements forefficient data presentation of relevant information. These disclosuresimprove diagnostic capability, monitoring and user efficiency.

It should be understood that the embodiments, examples and descriptionshave been chosen and described in order to illustrate the principals ofthe invention and its practical applications and is not intended to beexhaustive or to limit the invention to the precise form disclosed.Modifications and variations of the invention will be apparent to thoseskilled in the art in light of the above teaching. The embodiments werechosen and described to explain the principles of the invention and itspractical application to enable others skilled in the art to best usethe invention in various embodiments and with various modificationssuited to the particular use contemplated. The scope of the invention isdefined by the claims, which includes known equivalents andunforeseeable equivalents at the time of filing of this application.

The following references are hereby incorporated by reference.

US Patent Documents

U.S. Pat. No. 7,084,128, Yerxa, et al., Method for reducing intraocularpressure

U.S. Pat. No. 7,050,615, Avinash, et al., Temporal image comparison

U.S. Pat. No. 7,015,907 Tek, et al. Segmentation of 3D medicalstructures using robust ray propagation

U.S. Pat. No. 6,771,736, Sabol, et al., Method for displaying temporalchanges in spatially matched images

U.S. Pat. No. 6,266,452, McGuire, Image registration method

US Patent Publications

2006/0030768 System and method for monitoring disease progression orresponse to therapy using multi-modal visualization

2006/0184014 Registration aid for medical image

2006/0119858 Enhanced optical coherence tomography for anatomicalmapping

2005/0238253 Image registration

Other Publications

Carpineto et al., Custom measurement of retinal nerve fiber layerthickness using Stratus OCT in normal eyes, European Journal ofOphthalmology, Vol. 15, No. 3, 2005.

Choma, M. A., et al., Sensitivity advantage of swept source and Fourierdomain optical coherence tomography, Optics Express, Vol. 11, Issue 18,pp. 2183-2189 (September 2003)

de Boer, J. F., et al., Improved signal-to-noise ratio inspectral-domain compared with time-domain optical coherence tomographyOptics Letters, Vol. 28, Issue 21, pp. 2067-2069 (November 2003)

Frangi, A., Niessen, W. J., Hoogeveen, R. M, van Walsum, T., andViergever, M. A., Model-based Quantitation of 3-D Magnetic ResonanceAngiographic Images, October 1999, IEEE Trans on Medical Imaging, Vol.18, No. 10, pp. 946-956

Gerig, G., Kubler, O., Kikinis, R. and Jolesz, F. A., NonlinearAnisotropic Filtering of MRI Data, IEEE Trans. Pattern Analysis andMachine Intelligence, Vol. 11, pp. 221-232, 1992.

Ishikawa, H., Stein, D. M., Wollstein, G., Beaton S., Fujimoto, J. G.,and Schuman, J. S., Macular Segmentation with Optical CoherenceTomography, Investigative Ophthalmology and Visual Science, June 2005,Vol. 46, No. 6, pp. 2012-2017 (2005)

Leitgeb, R. A., et al., Performance of fourier domain vs. time domainoptical coherence tomography, Optics Express Vol. 11, No 8, pp. 889-894;

Maurer, C. R., Jr., and Fitzpatrick, J. M. 1993. A review of medicalimage registration. In Interactive Image-Guided Neurosurgery (R. J.Maciunas, Ed.), pp. 17-44.

Pal, Nikhil R. and Pal, Sankar K., A review on Image SegmentationTechniques Pattern Recognition, Vol. 26, No. 9, pp. 1277-1294. 1993.

Perona, P., Malik, J., Scale-Space and Edge Detection Using AnisotropicDiffusion, IEEE Trans. Pattern Analysis and Machine Intelligence, July1990, Vol. 12, No. 7, pp. 629-639

Sato, Y., Nakajima, S., Shiraga, N., Atsumi, H., Yoshida, S., Koller,T., Gerig, G., and Kikinis, R., Three dimensional multiscale line filterfor segmentation and visualization of curvilinear structures in medicalimages. Medical Image Analysis, 2(2):143-168, June 1998.

Vermeer, K. A., Vos, F. M., Lemij, H. G., Vossepoel, A. M., 2003. Amodel based method for retinal blood vessel detection. Computers inBiology and Medicine Vol. 34 (2004) 209-219

Wojtkowski, M., Bajraszewski, T., Gorczyliska, I., Targowski, P.,Kowalczyk, A., Wasilewski, W., Radzewicz, C., 2004 Ophthalmic Imaging bySpectral Optical Coherence Tomography, American Journal ofOphthalmology, Vol. 138, Iss. 3, pp. 412-419

Yu, Y., Acton, S., Speckle reducing anisotropic diffusion, IEEE Trans.On Image Processing, November 2002, Vol. 11, No. 11, pp. 1260-1270

Zana, F., and Klein, J.C. , A Multimodal Registration Algorithm of EyeFundus Images Using Vessels Detection and Hough Transform, IEEETransactions On Medical Imaging, Vol. 18, No. 5, May 1999

We claim:
 1. A method for propagating modifications to the segmentationof optical coherence tomography (OCT) volume data comprising: a.acquiring OCT volume data; b. segmenting the data to identify a surfacewithin the OCT volume; c. displaying a first slice of the OCT volumedata which includes at least a portion of the surface; d. accepting usermodifications of the surface at one or more points with respect to thedisplayed first slice; and e. automatically modifying the segmentationover a propagation zone containing additional portions of the volumedata based on the user modification to the first slice.